Repository navigation
Report full formulas for unit metadata overrides - #276
Conversation
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The bounded fix matches the stated requirements, includes focused regression coverage, and has no unresolved findings.
Review effort: Balanced
Findings: None
What changed in this PR
Fixes temperature override reporting in the shared unit audit without changing dataset values.
Changes:
- Reports different temperature zero points as
offsetmismatches with reference values. - Updates the schema and documentation while preserving existing sidecar imports.
- Adds core, CLI, and web regression coverage.
| File | Description |
|---|---|
| tests/unit/test_variable_unit_validation.py | Tests offset reporting, scale behavior, and sidecar compatibility. |
| tests/unit/test_variable_unit_interfaces.py | Tests CLI temperature override output. |
| tests/integration/test_inspector_metrics.py | Tests offset reporting through the web route. |
| docs/source/variable_unit_validation.rst | Documents offset mismatches and schema compatibility. |
| docs/source/_static/variable-unit-metadata.schema.json | Allows the offset mismatch kind. |
| aidrin/structured_data_metrics/variable_unit_validation.py | Detects temperature offsets using explicit Pint quantities. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
houjun
changed the title
jeanbez
left a comment
jeanbez
left a comment
There was a problem hiding this comment.
Thank you for tackling this. Reporting the temperature overrides as offsets is the right fix: on develop, a degC to kelvin override is reported as a "274.15× scale difference", which is simply wrong, and the new formulas make the mismatch much easier to understand.
However, we found a few regressions when running the PR against develop (tests pass, 1489 passed, and flake8 is clean, but none of the cases below are covered). We reproduced each of them through calculate_variable_unit_validation with a user override in the sidecar:
| Override | develop |
This PR |
|---|---|---|
cmH2O to kPa |
scale, factor 0.0980665 |
raises KeyError: 'kg' |
dimensionless to radian |
scale, factor 1.0 |
conversion, factor None |
count to dimensionless |
scale, factor 1.0 |
conversion, factor None |
Details are in the inline comments. Could you also add a test for each case (no exception for cmH2O to kPa, factor 1.0 kept for dimensionless to radian, and the expected display value for dimensionless units), so they stay fixed?
Related Issues / Pull Requests
Closes #275. Follow-up to #258.
Description
Overriding detected units previously treated Celsius-to-Kelvin as a 274.15× scale difference, and the initial offset fix reported only a zero point for Celsius-to-Fahrenheit. The shared audit now reports the complete formula, such as
°F = °C × 1.8 + 32, for any supported Pint scale/offset conversion regardless of the variable name or physical dimension. Multiplicative conversions also show their direction explicitly, such asg/m³ = kg/m³ × 1000.The slope comes from Pint's definitions, avoiding cancellation when an offset is large relative to the scale. Rounded coefficients use
≈; reverse Fahrenheit-to-Celsius reports°C ≈ °F × 0.555556 - 17.7778. Logarithmic/unknown converters, unsupported conversions, and numerically unrepresentable coefficients report that the formula is unavailable instead of claiming a scale or affine relationship.Dataset values and detected observations remain unchanged. Scale mismatches retain their numeric
conversion_factor; offset mismatches and the new genericconversionkind use null. Existing version-1 sidecars remain importable, with unchanged fields and version. Consumers validating against a cached schema must refresh it for the addedoffsetandconversionenum values. Dimension and classification mismatches retain their behavior. The web, CLI, and library share this reporting path.What changes are proposed in this pull request?
Checklist:
Validation
git diff --check: passed.Coverage includes both directions of Celsius/Kelvin and Celsius/Fahrenheit; pressure, speed, density, time, length, and dimensionless scaling; non-temperature offset definitions; small slopes with large offsets; rounding and scientific notation; logarithmic units and their aliases; unavailable conversions and numerical range limits; schema validation, legacy/new sidecar round trips, preserved source values/observations, CLI output, and the web route.